Papers with LSTM language models
Are All Languages Equally Hard to Language-Model? (N18-2)
Copied to clipboard
| Challenge: | a fair comparison of language models is tricky because of the size of the corpora and the variability of orthographic systems. |
| Approach: | They propose a framework for fair cross-linguistic comparison of language models . they show that in some languages, textual expression is harder to predict with n-gram models compared to LSTM models based on translated text . |
| Outcome: | The proposed framework is based on translated text and language models on 21 languages. |
Class-based LSTM Russian Language Model with Linguistic Information (2020.lrec-1)
Copied to clipboard
| Challenge: | LSTM models can be used in speech recognition systems at N-best or lattice rescoring stage. |
| Approach: | They propose to use word frequency and linguistic information to generate class-based LSTM Russian language models with various numbers of classes. |
| Outcome: | The proposed models outperform word-based models and word2vec models in terms of perplexity, training time, and word error rate. |
Using surprisal and fMRI to map the neural bases of broad and local contextual prediction during natural language comprehension (2021.findings-acl)
Copied to clipboard
| Challenge: | a prior work using surprisal only considered within-sentence context, using n-grams, neural language models, or syntactic structure as conditioning context. |
| Approach: | They extend the surprisal approach to use broader topical context . they identify distinct patterns of neural activation for lexical surprised and topical surpresed . |
| Outcome: | The proposed method captures effects of local and topical contexts on processing . it shows that local and broad contextual cues recruit different brain regions . |
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (N19-1)
Copied to clipboard
| Challenge: | State-of-the-art LSTM language models learn sequential contingencies with some success . LS models fail to learn other non-local grammatical dependencies, however . |
| Approach: | They compare LSTM language models with RNNGs to examine grammatical dependencies . they find that hierarchical supervision improves learning of non-local dependencies. |
| Outcome: | The proposed model outperforms the existing model on non-local dependencies and learns many of the Island Constraints on the filler-gap dependency. |
TF-LM: TensorFlow-based Language Modeling Toolkit (L18-1)
Copied to clipboard
| Challenge: | Existing deep learning tools offer building blocks but training and building models takes time and knowledge. |
| Approach: | They propose to make available LSTM language models trained on Dutch texts and English benchmarks. |
| Outcome: | The proposed model can be used to test the perplexity, predict the next word(s), re-score hypotheses or generate debugging files for interpolation with n-gram models. |